Skip to content

bundle: record and read deployment state via DMS - #6094

Open
shreyas-goenka wants to merge 22 commits into
mainfrom
isaac/pr6052-fixes
Open

bundle: record and read deployment state via DMS#6094
shreyas-goenka wants to merge 22 commits into
mainfrom
isaac/pr6052-fixes

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds code to read and write state using DMS, behind DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY.

Design decisions:

  1. The version is only created after a user approves a deployment. Not before.
  2. CreateVersion stages one operation per planned resource, so the CLI only ever calls UpdateOperation — there is no CreateOperation call. The service creates each staged operation as PENDING at sequence_id = 0, which is the precondition the CLI uses for its first update of a resource. Needs databricks-eng/universe#2420238 (merged).
  3. Jobs and pipelines are stamped with deployment ID and version ID.

Also adds an env var to toggle DMS, and records the API status and error code with a failure.

Testing Strategy

The whole acceptance suite runs a second time with recording on (EnvMatrix.DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY), so every bundle test exercises DMS against the local test server and asserts the same golden files either way — the nostamp helper strips the deployment stamp for that. Focused coverage of the recorded calls themselves lives under acceptance/bundle/dms.

Before private preview we will:

  1. Run these tests on cloud on our production workspaces.
  2. TBD: Run these tests on staging workspaces as well as part of our test infra.

What is missing?

  1. Serialized plan does not work yet. Will be fixed in a followup PR.
  2. Issues were reported in bugbash where applies would be state or plans would not be correctly computed. This could be because of caching in the service which we are fixing with strong reads. To be investigated later in any case.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: b4d23d2

Run: 33882675175

Env ❌​FAIL 🟨​KNOWN ✅​pass 🙈​skip Time
❌​ aws linux 27 1 275 52 5:58
❌​ aws windows 27 1 277 50 5:03
❌​ azure linux 27 1 274 52 6:59
❌​ azure windows 27 1 276 50 6:04
❌​ gcp linux 27 1 275 52 6:08
❌​ gcp windows 27 1 277 50 4:45
28 interesting tests: 27 FAIL, 1 KNOWN
Test Name aws linux aws windows azure linux azure windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/dms/declined-deploy ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/declined-deploy/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/depends-on ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/depends-on/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/deployment-metadata-change ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/deployment-metadata-change/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/emptied-resource ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/emptied-resource/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/existing-state ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/existing-state/DATABRICKS_BUNDLE_ENGINE=direct/DMS= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/multiple-resources ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/multiple-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/no-drift ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/no-drift/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/no-resources ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/no-resources/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/provenance ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/provenance/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/record ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/record-failure ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/record-failure/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/record/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/stale-plan ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN= ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/stale-plan/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true/READPLAN=1 ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/successful-recreate ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/dms/successful-recreate/DATABRICKS_BUNDLE_ENGINE=direct/DMS=true ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F

Comment thread bundle/direct/bundle_apply.go Outdated
// of and the next deploy would create them a second time. Checked here rather
// than only where operations are recorded, which is after the resource has
// already been modified.
if err := opQueue.firstErr(); err != nil {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could eventually extend this to record and return all multiple errors that happened.

@@ -0,0 +1,4 @@

=== An operation upload failure fails the deploy instead of reporting only at the end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's hard to make a assert more here because we cannot control how many requests went through. We could harden this test by making the number of workers configurable and 1. Omitting for now.

Comment thread acceptance/bundle/dms/test.toml Outdated
@@ -0,0 +1,18 @@
Local = true
Cloud = false

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll rollout the service to our test env by next week at which point we can start turning these tests on on cloud as well.


# Deployment Metadata Service (DMS) recording is only supported by the direct
# engine; it is a no-op on terraform.
EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["direct"]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually we can also make DMS part of the default test matrix - to get more coverage - this can be rolled out slowly.

Comment thread bundle/direct/opqueue_test.go Outdated
assert.Empty(t, f.recorded())
}

func TestOperationQueueCloseIsIdempotent(t *testing.T) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not strictly necessary given we only have one close site.

@shreyas-goenka
shreyas-goenka marked this pull request as ready for review July 29, 2026 15:22
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/bundle/ - needs approval

470 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @andrewnester, @lennartkats-db, @anton-107

/bundle/ - needs approval

26 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @andrewnester, @lennartkats-db, @anton-107

/cmd/bundle/ - needs approval

Files: cmd/bundle/generate/dashboard.go, cmd/bundle/generate/genie_space.go, cmd/bundle/utils/process.go
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @andrewnester, @lennartkats-db, @anton-107

General files (require maintainer)

23 files changed
Based on git history:

  • @denik -- recent work in bundle/direct/, bundle/phases/, libs/testserver/

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.

Comment thread libs/dms/recorder.go Outdated
return &apiVersionCreator{client: c}
}

func (a *apiVersionCreator) CreateVersion(ctx context.Context, deploymentID, versionID string, body createVersionRequest) (*bundledeployments.Version, error) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will be replaced with SDK calls soon in a followup.

Comment thread libs/testserver/bundledeployments.go Outdated
// The extra field cannot be added by embedding Deployment in a wrapper struct:
// Deployment has its own MarshalJSON, which is promoted to the wrapper and
// silently drops any sibling field.
func deploymentBody(d *dmsDeployment) (map[string]any, error) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goes away on a SDK bump

Comment thread cmd/bundle/summary.go Outdated
Comment thread libs/dms/recording_test.go Outdated
r := NewRecorder(RecorderOptions{Service: f, Versions: fakeVersions{requests: &f.versions}, DeploymentID: "stored-id", StatePath: testStatePath, Metadata: Metadata{TargetName: "dev", DisplayName: testDisplayName}, VersionType: VersionTypeDeploy})

err := r.CreateVersion(t.Context())
assert.ErrorContains(t, err, "internal error: no deployment found for the file with object id stored-id")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should never happen if the service is working properly, since htis indicates that the file exists but there was no deployment returned by GET deployment.

Comment thread bundle/direct/bundle_apply.go Outdated
// (sv.Value) come from the write just performed; GetResourceID reads
// the ID assigned by Deploy. depends_on is recorded alongside the config
// because it cannot be recomputed from it (see dstate.RecordedState).
if err := opQueue.record(ctx, resourceKey, action, b.StateDB.GetResourceID(resourceKey), sv.Value, d.DependsOn); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong, this records "recreate" once, correct?

We discussed sending two requests - one for Delete and one for subsequent Create.

We also discussed moving recording hooks inside SaveState, so that we record the event ASAP. I'm considering for some resources to record state earlier and/or multiple times, by moving recording into SaveState we'll be able to handle that with DMS #5391

Comment thread bundle/direct/opclient.go Outdated
"github.com/databricks/databricks-sdk-go/service/bundledeployments"
)

// The CLI calls the operations API directly rather than through the generated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to fix this. I'll investigate in a followup.

Comment thread libs/dms/recording.go Outdated

// The server expires a version's lease if it does not receive a heartbeat
// within a 2-minute TTL; we heartbeat well inside that window.
const defaultHeartbeatInterval = 30 * time.Second

@shreyas-goenka shreyas-goenka Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parts of the code here should be removed and replaced by the SDK soon.

Comment thread bundle/phases/destroy.go Outdated
// node under the state directory, so files.Delete removes it and any later call
// fails with 404. CompleteVersion is idempotent, so the deferred call in Destroy
// is a no-op after this.
if err := recorder.CompleteVersion(ctx, true); err != nil {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its a bit of a chicken and egg problem. We can improve this by first cleaning up worksapce.file_path + artifact_path. Then completing the version and then deleting state_path + root_path.

Left as a followup exercise, the current PR is already big enough in scope.

Comment thread acceptance/bundle/test.toml Outdated
# unset; `deploy --plan` then creates the resources without it and the next plan reports
# drift. Stamping at plan time would mean `bundle plan` creating the deployment record,
# which is a design decision, so the saved-plan path is left out of the DMS run for now.
EnvMatrixExclude.dms_no_readplan = ["DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY=true", "READPLAN=1"]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll fix this in a followup.

Comment thread bundle/direct/oprecorder.go Outdated
// failureFields. A failure that arrives before any operation exists still
// goes through CreateOperation below, carrying the prior state.
fields = failureFields
body = updateOperationRequest{

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By omitting state here we ensure that the latest state that was uploaded in a partial operation is retained.

@shreyas-goenka
shreyas-goenka requested a review from denik August 11, 2026 12:46
varundeepsaini pushed a commit to varundeepsaini/cli that referenced this pull request Aug 17, 2026
…lates (databricks#6220)

## Changes

`bundle init` now writes `experimental.record_deployment_history: true`
into the generated `databricks.yml` when
`DATABRICKS_BUNDLE_INIT_RECORD_DEPLOYMENT_HISTORY` is set. It covers
every built-in template that generates a `databricks.yml`, and is off by
default so template output is otherwise unchanged.

## Why

Lets new projects opt into recording deployment history at creation time
instead of editing the config by hand afterwards. The variable is
temporary and deliberately scoped to `bundle init`: it persists the
setting in the generated project rather than toggling it for the bundle
being run.

Note the setting is still gated off (databricks#6094), so a generated project also
needs `DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY` to
validate until that gate is lifted. The acceptance test sets both and
runs `bundle validate`, so it will fail if the two ever drift apart.

## Tests

New acceptance test: all five templates with the variable on, plus the
default-off and explicitly-false cases.
Comment thread libs/dms/client.go
Service bundledeployments.BundleDeploymentsInterface

// raw sends what the generated client cannot; see requester.
raw requester

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will go away once we get everything in the SDK

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

general comment: please add such comments directly to code, otherwise they will be lost.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: What's still missing from SDK?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

q: What's still missing from SDK?

A couple of that need to be propogated to the SDK from the API spec. Some are still marked as development and need to be marked as PrPr.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's mark it as PrPr and use generated client?

@shreyas-goenka
shreyas-goenka force-pushed the isaac/pr6052-fixes branch 7 times, most recently from 47e54da to 766a902 Compare September 3, 2026 21:37
shreyas-goenka and others added 9 commits September 4, 2026 01:55
Record each deploy's resource state to the deployment metadata service and
read it back, making the service the source of truth for a recorded bundle:
the local state file becomes a tombstone and a leftover WAL is discarded.
Gated on experimental.record_deployment_history (direct engine only), and
supports deploy --plan.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…anged

Co-authored-by: Isaac <no-reply@databricks.com>
…source

Co-authored-by: Isaac <no-reply@databricks.com>
…, drop dedicated summary test

Co-authored-by: Isaac <no-reply@databricks.com>
…op redundant operation-upload-message test

Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka and others added 13 commits September 4, 2026 09:30
…pen reads the workspace client from context

Co-authored-by: Isaac <no-reply@databricks.com>
…hem from DeploymentBundle

The phases now reach DMS through StateDB (DmsClient, OperationBuffer, TakeOperationBuffer)
rather than DeploymentBundle mirror fields. The operation drain stays before CompleteVersion.

Co-authored-by: Isaac <no-reply@databricks.com>
…ffer in StateDB

- add dstate.StorageBackend (WORKSPACE_FILESYSTEM/DEPLOYMENT_METADATA_SERVICE), set by Open from the state marker; the plan carries its string form so deploy --plan can reject a plan whose target switched backends
- StateDB owns the recording lifecycle: RecordingError() and CompleteVersion() (idempotent via versionCompleted), with the operation-buffer drain moved into Finalize; drop the exposed OperationBuffer/TakeOperationBuffer getters and the recorder() indirection
- gate DMS behavior on StorageBackend()==DEPLOYMENT_METADATA_SERVICE / ConfiguresDeploymentHistory instead of DmsClient()!=nil
- consolidate plan validation in validatePlan (config-vs-plan backend match + stale-plan guard)

Co-authored-by: Isaac <no-reply@databricks.com>
…ate unbind, restore v3 read

- rename StampDeploymentID -> StampDeploymentIdForFirstVersion (only used for the first version's id)
- inline RecordDeploymentHistoryEnv into RecordsDeploymentHistory (single predicate)
- unbind: drop the config-based phase check; DeploymentBundle.Unbind opens without recording, so the state marker (not the config flag) refuses a recorded deployment
- restore forward-compat read of featureStateVersion=3: this CLI still writes v2, but reads a v3 state with no features and refuses one that carries a feature

Co-authored-by: Isaac <no-reply@databricks.com>
Extend declined-deploy, depends-on and deployment-metadata-change to capture the
serialized 'bundle plan -o json' (now carrying storage_backend), the DMS
list-resources view, and the raw recorded state. deployment-metadata-change pins
state_path, which print_state.py cannot resolve, so it relies on list-resources.

Co-authored-by: Isaac <no-reply@databricks.com>
…ate across the suite

Extend no-resources, multiple-resources, record, successful-recreate, no-drift,
stale-plan, emptied-resource and provenance to capture the serialized
'bundle plan -o json' (now carrying storage_backend), the DMS list-resources view,
and the raw recorded state at each steady-state point. no-drift shows only the
updated resource bumping to version 2; successful-recreate shows the recreate at
version 2; provenance masks the git commit.

Co-authored-by: Isaac <no-reply@databricks.com>
…-version golden

- deploy.go: StorageBackend() asserts the state is open, but only the direct engine
  opens it, so gate the DMS deployment-create block on stateEngine.IsDirect() first.
  Fixes a panic ("DeploymentState must be opened first") on terraform deploys, which
  the direct-only dms tests did not exercise.
- future_version: supported state version is now 3 (v3 is read-forward-compatible),
  so the too-new message reports 3.

Co-authored-by: Isaac <no-reply@databricks.com>
…version on recording error

- C11: StateDB holds DeploymentID/LatestVersionID (set at Open from the service); CalculatePlan
  stamps the plan's lineage from StateDB instead of the bundle config tree (config.History stays
  only for 'bundle summary'). Behavior-neutral.
- unbind: Open returns a sentinel ErrUnsettingRecording; DeploymentBundle.Unbind maps it to an
  unbind-specific message instead of the deploy/destroy-flavored guard text.
- CompleteVersion now completes as Failure when the drained buffer holds a recording error, even
  if the caller passed success=true, so a destroy whose uploads failed keeps its deployment record.

Co-authored-by: Isaac <no-reply@databricks.com>
… the readplan option

The dedicated readplan test duplicated stale-plan, which already deploys from a saved raw
plan under recording. Delete it and add the standard READPLAN matrix to stale-plan so its
initial deploy runs both from the saved plan and recomputed - the deploy is untraced so the
variant-dependent command stays out of the golden. Recorded operations still carry the
deployment_id/version_id lineage (asserted in depends-on and others).

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Extend the failure-injection tests to show how DMS records a failed or refused deploy:
failed-delete/recreate/update, failed-update-permissions, operation-upload-fails-once,
record-failure, version-never-created (and a WORKSPACE_FILESYSTEM plan on existing-state's
first deploy). Each captures the healthy recorded state and the recorded result of the failure.

Co-authored-by: Isaac <no-reply@databricks.com>
… wording

- rename StartRecording -> InitializeOperationBuffer (it opens the buffer; kept separate from Open
  because a first deploy's id does not exist until after approval)
- collapse the duplicated deployment/version fields to one public pair (DeploymentID/LatestVersionID);
  InitializeOperationBuffer records the first-deploy id, CompleteVersion derives the version from it
- RecordingError is now a no-op unless the backend is the deployment metadata service, keeping the
  non-recording path cleanly separate (like RecordFailure)
- stop surfacing the internal "deployment metadata service" name in user-facing errors; say
  "deployment history feature enabled" / "the deployment history service"
- drop the "in place of nil client checks" comment and restore the original featureStateVersion
  wording (minimal C12 diff)

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants